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ABSTRACT 

A general method for recovering missing DCT coefficients in 
DCT-transformed images is presented in this work. We model the 
DCT coefficients recovery problem as an optimization problem and 
recover all missing DCT coefficients via linear programming. The 
visual quality of the recovered image gradually decreases as the 
number of missing DCT coefficients increases. For some images, the 
quality is surprisingly good even when more than 10 most significant 
DCT coefficients are missing. When only the DC coefficient is miss- 
ing, the proposed algorithm outperforms existing methods according 
to experimental results conducted on 200 test images. The proposed 
recovery method can be used for cryptanalysis of DCT based selec- 
tive encryption schemes and other applications. 

1. INTRODUCTION 

As a sub-optimal de-correlation transform, the discrete cosine trans- 
form (DCT) [1| is one of the most widely used transforms in sig- 
nal and image processing applications, especially lossy audio, im- 
age and video compression. Many mainstream multimedia coding 
standards, such as JPEG, MPEG- 1/2/4 and H.264/AVC, are built on 
top of DCT. To reduce computational complexity, DCT is normally 
performed on N x N blocks for digital images and videos. Since 
DCT is a good de-correlation transform, all non-zero quantized DCT 
coefficients in each image block are coded to allow recovering the 
original image/video with a desired level of visual quality. The re- 
maining correlation between adjacent image blocks is normally han- 
dled by differential coding of DC coefficients or more complicated 
intra predictive coding mechanisms l2l . 

Among all DCT coefficients, the first one (called the DC coef- 
ficient) contains the most energy as well as perceptual information 
of an image block. Roughly speaking, the further a DCT coefficient 
is from the DC coefficient, the less perceptual information is repre- 
sented by it. Since DCT coefficients of each image block are largely 
uncorrected, all DCT coefficients of a block can be considered as 
separate "quality layers" with different significance. By exploiting 
this fact, some researchers proposed to implement perceptual en- 
cryption of digital images and videos by selectively encrypting part 
of DCT coefficients [3]|4][5][6]. F° r instance, Li et al. (4) proposed 
to encrypt DC coefficients and sign bits of other DCT coefficients 
(called AC coefficients) to conceal the rough and detailed views of 
MPEG-encoded video sequences, respectively. 
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Since DCT is an orthogonal transform, selective encryption of 
DCT coefficients was believed to be secure until Uehara et al. Q 
reported that missing DC coefficients can be approximately recov- 
ered from AC ones. The method proposed in [7| was improved by 
Li et al. 0, which reduces propagation errors and adopts a better 
estimate of the first DC coefficient via under-/over-flow rate mini- 
mization (FRM). Both DC recovery methods are based on two com- 
mon properties of digital images. First, there is strong correlation 
between neighboring pixels. Second, a smaller range of the DC co- 
efficient of a block can be calculated from the AC coefficients of 
the same block. Besides the above two DC recovery methods, Said 
l|9) developed a method to recover selectively encrypted AC sign 
bits, where a low-quality version of the original image is required 
to support the recovery process. None of the previous work can be 
easily generalized to handle the case when both the DC coefficient 
and some AC coefficients are encrypted. It remains an open question 
whether a combination of DC/ AC encryption is still secure. 

In this work, we propose an algorithm to recover an arbitrary 
set of missing DCT coefficients (except for the case when all DCT 
coefficients are missing). It offers a generic framework in which 
the DC recovery problem becomes a special case. To the best of 
our knowledge, this is the first time that a solution is developed to 
tackle such a generic and challenging problem. Simply speaking, 
we model the DCT coefficients recovery problem as an optimization 
problem and use linear programming to solve it. We have conducted 
experiments on 200 test images and shown that the proposed algo- 
rithm outperforms two existing DC recovery methods given in 171181 
significantly and consistently. The newly proposed algorithm does 
not depend on a low-quality version of the original image. Instead, 
it attempts to recover missing DCT coefficients solely from the in- 
formation contained in known DCT coefficients. Thus, it does not 
suffer the limitation reported in 1 9 ] . Although the DCT coefficients 
recovery problem is studied in the context of selective encryption, 
the proposed method for solving the problem is actually application- 
independent and may find potential applications in related areas such 
as image/video compression and error concealment. 

The rest of this paper is organized as follows. The proposed 
method is described in Sec. [2 Experimental results of an imple- 
mentation of the proposed method are reported in Sec. [3] where we 
compare its performance with those obtained by methods in Q [8) 
for performance benchmarking. Concluding remarks and future re- 
search directions are given in Sec.|4] 

2. PROPOSED METHOD 

In this section, we first discuss the modeling of the DC recovery 
problem after a brief introduction to two existing DC recovery meth- 
ods. Then, its generalization to the recovery of an arbitrary set of 
missing DCT coefficients is addressed. After setting up the mathe- 
matical model, we will study how to solve the optimization problem 



effectively in terms of low space and time complexity. 
2.1. Basic Problem: DC Recovery 

The two DC recovery methods proposed in [U are based on the 
following two properties. 

Property 1: The difference between any two neighboring pixels is a 
Laplacian variate with zero mean and a small variance. 
Property 2: For each block, the range of pixel values calculated 
from AC coefficients constrains the value of the DC coefficient. 

Property 1 is a well-known feature of most natural images while 
Property 2 is a result of the relationship between the pixel val- 
ues {x(i,j) £ [x m in, x m ax]}o<2,j<A^-i and the DCT coefficients 
{y(k, l)}o< k ,i<N-i as defined by N x N 2-D DCT itself: 

x ^ti = I^o< fc ^<iv-i A( '' j ' A: ' /) ' 2/(/c ' /) ' (1) 

= "X?2/(0, 0) + ^ <m<at-i A(iJ,k,l)y(k,l), 

iV (fc, o^(o,o) 

where 

A(m,*,0 = C(k)C(l) cos (£±*2*2[) cos (^±-£*) , 

C(k) = y/l/N when k = and when k > 0. In above, 

the DC coefficient, denoted by y(0, 0), is obviously constrained by 
the sum term, which is DC-free, and [ 1. Note also that 

Eq. (|T|) is a linear map and the indices are relative to each block. We 
will use x = A • y to denote the block- wise DCT below. 

The DC recovery method in (7) scans the whole image block 
by block and tries to align all DC coefficients to minimize the sum 
of differences of pixel pairs along each block boundary. Property 2 
is used to estimate the global intensity of the whole image. Main 
drawbacks of this method include large propagation errors and a less 
accurate estimate of the global intensity. The improved DC recov- 
ery method in (8) introduces online pixel under-/over-flow removal 
in the scanning process and an under-/over-flow rate minimization 
process to estimate the global intensity more accurately. While the 
improved method can produce fairly good results for many images, 
the quality of recovered images is not always high. This can be ex- 
plained by the fact that the scanning process is essentially local in 
the sense that only two neighboring blocks are considered at any 
time and cannot always lead to the global optimum solution. 

Actually, the DC recovery problem can be formally modeled as 
an optimization problem w.r.t. some measure on the reconstructed 
pixel values and be solved to get the global optimum solution. We 
use variables x(i,j) and y(k, I) to denote the value of pixel 
and the DCT coefficient (h,l), respectively. Then, the DC recovery 
problem can be written as 

minimize f ({ x (h j)}o<i,j<N-i) 

subject to x = A • y, 

Xmin ^ j) ^ ^max 5 (2) 

y(k, I) — y*(k, I) for all AC coefficients. 

The last equality fixes AC coefficient y(k,l) to its known value 
y*(k,l) while DC coefficient 2/(0,0) of each block remains to be 
a variable. Note that the bounds on x(i,j) and the transformation 
constraints imply lower and upper bounds on variable y(0, 0). 

The objective /(•) may be any convex function to allow a 
polynomial-time optimization algorithm provided that / can be 
evaluated in polynomial time. Because of Property 1, we choose 
^2{(i,j),(i',jf)}\ x (.^j) ~ x (i'>f)\> where the sum ranges over all 



pairs of neighboring pixels (z, j) and (i f ,f). Moreover, this choice 
makes the optimization problem a linear one, which can be solved ef- 
ficiently. The linearization is done by introducing variables hij^j/ 
for each term of the sum and the optimization problem becomes 

minimize S , j , *' ,j' 

subject to x(ij) - x(i',f) < hij^jr, (3) 
x(i',j') - x(i,j) < hijjjt, 
and the constraints of Eq. ©. 

We can derive the implied bound < hij^j/ < x max — £min 
accordingly. Furthermore, variable hi^y jt will be tight at \x(i, j) — 
x(i f ,f)\ for the optimum solution. Since the DC value of a block 
contributes to all pixels of the same block equally, only the pairs 
of neighboring pixels that belong to different blocks are relevant. 
Hence, we may restrict ourselves to these pairs. 

The above optimization problem contains a free variable corre- 
sponding to the global intensity when the dynamic range of the op- 
timum solution is smaller than [x m i n , x max ]. In this case, the global 
intensity can be shifted without changing the value of the objective. 
In other words, there are multiple optimum solutions. We handle 
this problem by shifting the histogram of the recovered image (i.e., 
one solution) towards the midpoint of [x m i n , x ma x] until the left and 
right margins of the histogram are equal. 

2.2. General Problem: Arbitrary DCT Coefficients Recovery 

The generalization from DC recovery to arbitrary DCT coefficients 
recovery is straightforward. That is, we can fix y(k, I) — y*(k, I) 
for known DCT coefficients while constrain all remaining unknown 
variables in a range [y m in(fc, I), 2/max(&, /)] depending on their lo- 
cations (i.e. spatial frequencies). Note that when at least one AC 
coefficient is unknown, pixel pairs inside each block shall not be 
discarded as done in the previous case. The above generalization 
clearly maintains the linearity of the optimization problem. Hence, 
we can use the linear programming (LP) technique to solve both 
the basic DC recovery problem and the general DCT coefficients re- 
covery problem. In our implementation, we adhere to the general 
problem and treat the DC recovery problem as a special case. 

2.3. Solving Optimization Problem 

From a theoretical point of view, linear programs can be solved in 
time bounded polynomially in the binary encoding length of the in- 
put data using the ellipsoid method 1 10] or the interior point meth- 
ods ifTTl . The latter yields an efficient implementation in practice. 
Other methods with good practical performance include the primal 
and dual simplex algorithm. We refer to |[T2l for interested readers. 

The LP solution consists of floating point values for the pixel 
values and DCT coefficients. After the optimization, we round the 
pixel values to fixed precision numbers, i.e. to integers. Our experi- 
mental results show that the rounding error is negligible in terms of 
visual quality, so we do not re-optimize after the rounding step. Ad- 
ditional integrality constraints for pixel values lead to Integer Linear 
Programming (ILP), which is NP-hard in general fT3l . Whether this 
also holds for our application is a topic for further research. As of 
yet, we know that the problem remains polynomial time solvable for 
the simpler case of DC-only recovery. The reason for this is that 
there is an equivalent formulation as an LP over an integer polyhe- 
dron for which ILP is not more difficult than LP IT4l Chapter 5.15]. 

2.4. Complexity Analysis 

Given annxm image and U unknowns in each ofBNxN blocks, 
the mathematical model presents a large-scale optimization problem 



with 2nm — (n + m) /i-variables and U • B ^/-variables. Since all 
other unknowns can be uniquely determined by these variables, they 
can be eliminated by a presolve step of the optimization process. For 
the basic DC recovery problem, the number of variables drastically 
reduces as only pixel pairs along block boundaries contribute to the 
objective value. Assuming that n and m can be divided by N, the 
number of variables involved is reduced to nm/N — (n + m) h- 
variables and nm/N 2 ^/-variables. 

The worst-case time complexity of the interior point method for 
solving LP problems with V variables and with a size of L bits is 
0(V 3 L) arithmetic operations |15|. Here, the problem size L is 
the total number of bits needed to store the whole LP including the 
objective, the constraint matrix and the right-hand side of the con- 
straints. For our problem, the value of L is 0(nmU) thanks to the 
sparsity of the constraint matrix. As a result, the worst-case time 
and space complexities are 0(n 4 m 4 U) and 0(nmU), respectively. 
Since our problem is rather sparse, the running time scales much 
better in practice as reported in the next Section. 

When the image size is large, the required memory and compu- 
tational time is untractable for low-end computers such as PCs. This 
problem can be partly solved by dividing the whole image into suf- 
ficiently small patches and the global intensity of different patches 
can be adjusted to minimize the discontinuity along patch bound- 
aries. It ia also possible to limit the scale of the problem by reducing 
the number of pixel pairs used in the model. Both measures will in- 
evitably compromise the visual quality of the recovered image. In 
this paper, we only consider images that can be directly handled by 
the optimization algorithm. The above "divide and conquer" idea 
will be a future research topic. 

3. EXPERIMENTAL RESULTS 

To validate the real performance of the proposed method for recover- 
ing multiple missing DCT coefficients, we built an implementation 
based on the commercial optimization software package IBM ILOG 
CPLEX 12.2 fl6l . More precisely, we used the constrained barrier 
optimizer (an interior point method) in CPLEX because it turned out 
to be the most suitable method for our purpose. 

We ran the proposed method on 200 test images and evaluated 
the subjective and objective visual quality of the recovered images 
against the original ones to see how well the proposed method 
works. The objective visual quality metrics include PSNR and nine 
other ones included in the MeTriX MuX Visual Quality Assessment 
(VQA) Package [17]. All the 200 test images are 8-bit gray-scale 
images, so x m in — and x max — 255. 

3.1. Performance of DC Recovery 

The performance of DC recovery is nearly perfect as judged the au- 
thors as human observers. For all 200 test images, we cannot see any 
obvious quality degradation other than some global intensity shift, 
which is rarely viewed as visual distortion since no perceptual in- 
formation is lost. The recovery results on two typical test images 
"Lenna" and "cameraman" are shown in Fig. Q] The DC recovery 
process is also very efficient. The whole process could finish in 
less than 10 seconds on all laptop/desktop PCs we tested in our ex- 
periments and the average running time over the 200 test images is 
around 5 seconds on our main test computer (a quad-core desktop). 

We compared the performance of the proposed method and the 
FRM method proposed in 1 8 ]. The results are shown in Fig.[2 where 
the x-axis denotes the image index and the y-axis shows the differ- 
ence of quality metrics of the two methods. A positive value means 
that the proposed method performs better. One can see that the pro- 
posed method outperforms the FRM method significantly and con- 




Fig. 1. The recovery results of two test images (a) "Lenna" and (d) 
"cameraman": (b) and (e) DC-free images by setting the missing DC 
coefficients to the midpoints of the valid ranges; (c) and (f) recovered 
images by the proposed method. 



sistently over the whole image database. For some VQA metrics, a 
few images seems to have worse quality, but a manual check by the 
authors revealed no noticeable difference in subjective quality. Since 
the FRM method outperforms the method proposed by Uehara et al. 
in 0, the proposed method is the best among the three. 



Pr[A(PSNR)>0] = 0.85, Mean(A(PSNR)) = 2.46242 



Pr[A(SSIM)>0] = 0.94, Mean(A(SSIM)) = 0.0221093 



Pr[A(MS-SSIM)>0] = 0.96, Mean(A(MS-SSIM)) = 0.0392099 



Pr[A(WSNR)>0] = 0.85, Mean(A(WSNR)) = 2.49078 
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Fig. 2. The performance comparison between the proposed method 
and the DC recovery method proposed in | 8 ], where the red dashed 
line shows the mean value of the difference of quality metrics. 



3.2. Performance of DCT Coefficients Recovery 

We also studied the performance of the proposed method when U > 
1 most significant DCT coefficients in each block are unknown. As 
shown in Fig. the visual quality of the recovery image gradually 
decreases while the value of U increases. This is expected since we 
have less and less known DCT coefficients available for recovering 
the missing one. The recovery performance is still good for some 
images even when more than 10 most significant DCT coefficients 
are missing as shown in Fig. [4])- The average running time ranges 
from around 5 seconds (U = 1) to around 10 minutes (U — 15). 
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Fig. 3. The average visual quality measured in terms of PSNR and 
SSIM of the 200 test images for U = 1, • • • , 15. 




Fig. 4. The recovery results when 15 most significant DCT coeffi- 
cients are missing: (a) the original image "cameraman"; (b) the im- 
age by setting all missing DCT coefficient to the midpoints of their 
valid ranges; (c) the recovered image by the proposed method. 



3.3. Actual Complexity 

We also made some experiments to check the actual time and space 
complexities of the optimization problem. To make the estimated 
time and memory usage more accurate, we limited the number of 
threads to 1 although parallelism is supported by CPLEX. Note that 
this does not affect the reconstructed images. When the number of 
unknowns U is fixed, the running time grows almost quadratically 
and the memory usage scales linearly in the number of pixels nm. 
For example, about 4-10 KB memory per pixel is used on average 
for U = 1, • • • , 15. When the number of pixels nm is fixed, both the 
running time and memory usage grows linearly in the number of un- 
knowns U. Therefore, we conclude that the actual time complexity 
is 0(n 2 m 2 U) and the actual space complexity is 0(nmU). 

4. CONCLUSION AND FUTURE WORK 

A method to recover multiple missing DCT coefficients in DCT- 
transformed images was proposed in this work. It was formulated as 
an optimization problem that can be effectively solved by linear pro- 
gramming. Experimental results validated the excellent performance 
of the proposed method. Besides its direct application to cryptanaly- 
sis of selective encryption schemes, the fact that U > 1 DCT coeffi- 
cients can be approximately estimated from other known coefficients 
drives us to re-think how much information of DCT coefficients is 
needed in encoding and transmission. 

In Sec. 12.41 we mentioned one future research topic on how to 
overcome the problem with large images. There are many other top- 
ics that deserve further investigation. They are given below. 

• More theoretical analysis about solvability, complexity of the 
problem and real performance of the proposed method. 



• Further improvement of the method to balance time/space 
complexity and the visual quality of the recovered image. 

• Generalization of the method to handle unknown sign bits of 
some DCT coefficients. 

• Generalization of the method to permuted DCT coefficients 
(i.e., when locations of some DCT coefficients are unknown). 

• Generalization of the method to color image and video by 
exploiting more correlation among different color channels 
and different video frames. 

• Generalization of the method to other transforms, especially 
DWT used in JPEG2000. 

• Applications of the method to image and video compression 
by coding less information about transform coefficients. 
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